Mobile device application for ocular misalignment measurement

ABSTRACT

Disclosed are a mobile device and method which include acquiring, by an image acquisition unit installed in a mobile device, an image of eyes of a patient while light provided by a light source reflects from an optical surface of the eyes of the patient; and obtaining, by a processor installed in the mobile device, ocular misalignment measurements, including a magnitude and a direction of ocular misalignment in the eyes of the patient, using the acquired image or set of images.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of priority under 35 U.S.C. § 119(e)to U.S. Provisional Patent Application No. 62/295,869, filed Feb. 16,2016 in the United States Patent and Trademark Office, which isincorporated herein by reference in its entirety.

FEDERALLY SPONSORED RESEARCH AND DEVELOPMENT

This invention was made with government support under R43 EY025902awarded by the National Institutes of Health. The government has certainrights in the invention.

TECHNICAL FIELD

The present disclosure relates generally to ocular misalignmentmeasurement, and more particularly, to a mobile device application forocular misalignment measurement.

BACKGROUND

Ocular misalignment, or, more commonly, eye misalignment, refers to avisual disorder in which the eyes are misaligned and point in differentdirections. When the eyes are misaligned, one eye will typically fixateon an object of interest while the other eye turns in another direction,such as inward (i.e., esotropia), outward (i.e., exotropia), down (i.e.,hypotropia), or up (i.e., hypertropia). Ocular misalignment, whichaffects about 3 to 8% of the United States population, can lead todouble vision, spatial confusion, and diminished depth perception ifleft untreated. Misalignment of the eyes in children can lead toamblyopia, which could lead to permanently reduced vision in one of theeyes (affecting about 4% of the population). Misalignment in adults canresult in double vision and difficulty in depth perception. People withneurological disease such as multiple sclerosis or injury such asstroke, trauma are at greater risk of developing strabismus, andchildren below eight years of age are at additional risk forunderdevelopment of vision due to strabismic amblyopia.

Heterophoria is a condition in which misalignment exists but iscompensated under normal viewing conditions, and so there is nostrabismus, or it is intermittent. The condition is revealed using testswhich occlude an eye (dissociation) and look for drifting of the eye out(exophoria), in (esophoria), up (hyperphoria), or down (hypophoria).Patients with heterophoria are typically characterized as having eyeswhich point in different directions while in a resting state (eyesclosed or in a dark environment). With such patients, the tendency ofthe eyes to deviate is kept latent by binocular fusion during typicalviewing conditions, and symptoms of double vision do not occur or areintermittent. People with heterophoria may experience eye strain andreduced visual endurance.

Strabismus is a condition that interferes with binocular vision bypreventing a person from directing both eyes simultaneously towards asingle fixation point. That is, the eyes are unable to gaze at once tothe same point in space. Strabismus usually results in normal vision inthe preferred sighting (or “fixating”) eye—the eye that the patientprefers to use—but may cause abnormal vision in the deviating orstrabismic eye. Due to the misalignment between the images beingprojected to the brain from the two eyes, the brain tends to suppressthe images from the deviated eye. Such misalignment may be constant orintermittent, and symptoms of strabismus include double vision, visualconfusion, incoordination, reduced vision in the deviated eye inchildhood onset, and higher risk of falls in the elderly.

Unless detected and treated, e.g., through corrective lenses, eyepatches, vision therapy, surgery, and the like, cases of ocularmisalignment can result in permanent problems with visual development.Early ophthalmologic diagnosis and treatment can substantially reducethe risk of developing permanent vision loss, such as amblyopia (i.e.,lazy eye), in the deviated eye. Effective treatment can allow fornormalization of vision development, realignment of the eyes, andrestoration of stereo or three-dimensional vision.

Ocular misalignment can be diagnosed and measured using severaldifferent techniques. The measurements help quantify a risk factor forthe patient developing an ocular misalignment condition, such asamblyopia. One technique, known as light reflex testing, or Hirschbergtesting, involves directing a patient to look at a point of light infront of the patient's face and observing where the light reflects offthe corneas. When performing the test, the ocular reflections (i.e.,light reflexes) on the corneal surface of both eyes caused by the lightare compared. In a patient with normal fixation, the ocular reflectionsin both eyes will be substantially symmetrical. In a patient exhibitingocular misalignment, the ocular reflections will be asymmetrical.

Notably, a photographic version of Hirschberg testing can be used toquantify the magnitude and direction of ocular misalignment in thepatient. In this case, a photograph of the patient's eyes is acquired,and characteristics of the patient's eyes are analyzed using theacquired photograph. Conventional clinical practice for ocularmisalignment measurement, however, can be tedious, subjective, andrequires significant amount of subject cooperation and doctor'sexpertise. Meanwhile, automated devices using photographic analysis,while helpful, are expensive, and limited to specialty clinics. Thus,the accessibility to convenient and inexpensive machines is currentlyhindered.

SUMMARY OF THE DISCLOSURE

The present disclosure provides for a mobile device application and amobile device which performs said mobile device application toautomatically measure ocular misalignment in the eyes of a patient byacquiring an image or a set of images of the patient using the mobiledevice and providing quantifiable measures of misalignment based on theacquired image. The application uses conventional mobile devicehardware, including a built-in camera and flash, to capture a photographof the eyes of a patient while light from the flash is shined in thepatient's eyes and reflects from a corneal surface thereof(alternatively, light may be provided by an external source). After aphotograph of the patient's eyes is captured, the applicationautomatically processes the photograph to measure the magnitude ofocular misalignment. The entire processing is performed locally on themobile device itself; thus, the application is not required to sendcaptured pictures to a remote server for processing. Based on themeasurements, magnitude of strabismus is measured and a risk of thepatient developing a condition, such as amblyopia, can be quantified andprovided to the patient.

In particular, a camera of a mobile device can be directed toward theeyes of the patient. The application can provide guidance to thepatient, examiner, or other user to adjust the distance between themobile device and the patient's face and the angle of the mobile devicecamera with respect to the patient's eyes. As the eyes of the patientfixate on a particular object, the mobile device can capture aphotograph of the patient's eyes while light from the built-in flash, orotherwise, reflects from an optical surface of the eyes (e.g. cornea).The application then performs a variety of calculations to measure theocular misalignment, including detecting the iris, pupil, limbusboundary, and corneal reflection, and then converting the detectedfeatures into the real space (e.g., physical measurements, mm or prismdiopters). Then, the magnitude of ocular misalignment can be calculatedand displayed for the reference of the patient, doctor, or other user.

According to embodiments of the present disclosure, a method includes:acquiring, by an image acquisition unit installed in a mobile device, animage of eyes of a patient while light provided by a light sourcereflects from an optical surface of the eyes of the patient; andobtaining, by a processor installed in the mobile device, ocularmisalignment measurements, including a magnitude and a direction ofocular misalignment in the eyes of the patient, using the acquiredimage.

The obtaining of ocular misalignment measurements may include: detectinga reflection on the corneal surface of a left eye of the patient and areflection on the corneal surface of a right eye of the patient usingthe acquired image; and comparing a position of the reflection in theleft eye to a position of the reflection in the right eye. The obtainingof ocular misalignment measurements may further include: detecting areference point in the left eye and a reference point in the right eyeusing the acquired image; calculating a first distance between aposition of the reference point in the left eye and the position of theocular reflection in the left eye; calculating a second distance betweena position of the reference point in the right eye and the position ofthe ocular reflection in the right eye; and calculating a differencebetween the first distance and the second distance.

Additionally, the obtaining of ocular misalignment measurements mayfurther include measuring the positions of the reflections in the leftand right eyes and the positions of the reference points in the left andright eyes in image space using the acquired image. The reference pointsin the left and right eyes may relate to a center of an iris or a pupilof the patient. The obtaining of ocular misalignment measurements mayfurther include converting the first distance and the second distanceinto degrees or prism diopters using a Hirschberg ratio and an internalcalibration factor based on iris diameter.

The method may further include diagnosing a condition of the patientbased on the ocular misalignment measurements. The diagnosis may beperformed according to the calculated difference between the firstdistance and the second distance. Further, the diagnosing of thecondition may include: comparing the calculated difference between thefirst distance and the second distance to a predetermined misalignmentthreshold; and determining the condition of the patient based on thecomparison of the calculated difference to the predeterminedmisalignment threshold. Also, the diagnosing of the condition mayfurther include diagnosing the patient with a condition associated withocular misalignment when the calculated difference is greater than orequal to the predetermined misalignment threshold.

The method may further include detecting an iridoscleral border and apupillary margin of the eyes of the patient using the acquired image.The iridoscleral border and the pupillary margin may be detected usingan adaptive thresholding algorithm.

In addition, the mobile device may be a smart phone or a tablet. Thelight source may be a flash producing device installed in the mobiledevice configured to produce a flash of light during the acquisition ofthe image. The light source can be independent of the mobile device.Further, the image acquisition unit may be a rear-facing camerainstalled in the mobile device. In the alternative, the imageacquisition unit may be a front-facing camera installed in the mobiledevice. The method may further include recognizing an input provided bya user to initiate the acquisition of the image of the eyes of thepatient using a rear-facing camera installed in the mobile device, wherethe user and the patient are not the same. Alternatively, the method mayfurther include recognizing an input provided by a user to initiate theacquisition of the image of the eyes of the patient using a front-facingcamera installed in the mobile device, where the user and the patientare the same.

The method may further include: acquiring a plurality of images of theeyes of the patient; determining an image quality of the acquiredplurality of images; and using an image having a highest image qualityamong the acquired plurality of images for the measuring of themagnitude of ocular misalignment in the eyes of the patient. Also, themethod may further include: acquiring a plurality of images of the eyesof the patient; obtaining the ocular misalignment measurements in eachof the acquired images; generating a composite of the obtained ocularmisalignment measurements; and determining an overall ocular measurementbased on the generated composite.

During the acquisition of the image of eyes of the patient, both eyes ofthe patient may fixate on a point located on the mobile device.Alternatively, one eye of the patient may fixate on an external objectwhile the other eye of the patient cannot view the external object dueto a positioning of the mobile device.

The method may further include: acquiring a plurality of images of theeyes of the patient, where the eyes of the patient fixate on a differenttarget for each of the plurality of acquired images; and obtainingocular misalignment measurements in the eyes of the patient in each ofthe plurality of acquired images. In this regard, the method may furtherinclude calculating a Hirschberg ratio of the patient based on theocular misalignment measurements. The method may further include:detecting an iris diameter and an inter-pupillary distance of thepatient based on the acquired image; and calculating the Hirschbergratio of the patient based further on the detected iris diameter andinter-pupillary distance of the patient. Also, the method may furtherinclude: assisting a user in positioning the mobile device to obtain apredefined angular change in position of the different targets relativeto the eyes during the acquisition of the plurality of images.

Additionally, the method may further include: measuring a kappa anglebetween an optical axis and a visual axis of the eyes of the patientbased on the acquired image; and diagnosing a condition of the patientbased further on the measured kappa angle. In this regard, during theacquisition of the image of eyes of the patient, one eye of the patientmay fixate on the light provided by the light source. The method mayfurther include diagnosing which eye of the eyes of the patient ismisaligned and a misalignment direction of the misaligned eye based onthe kappa angle. Further, the method may include measuring a fixationreliability index of the non-misaligned eye based on a deviation of thenon-misaligned eye from the kappa angle. Even further, the method mayinclude calculating an accommodative convergence/accommodation (AC/A)ratio of the eyes of the patient based on the acquired image.

The method may further include compensating for a rotation of the mobiledevice when the mobile device is rotated during the acquisition of theimage. Measurements obtained by one or more motion sensors installed inthe mobile device may be used for the compensation of rotation of themobile device.

Moreover, the method may further include detecting features in the eyesof the patient including a pupil, an iris, a limbus boundary, and areflection on the optical surface using the acquired image. In thisregard, the method may further include measuring positions of thedetected features in the eyes of the patient in image space according tothe acquired image. The method may further include detectingintermittent ocular misalignment motions in the eyes of the patient.

Additionally, the method may further include displaying information on adisplay screen of the mobile device for assisting in the acquisition ofthe image of eyes of the patient. The method may further includedisplaying a graphic on a display screen of the mobile device forassisting in aligning the eyes of the patient with the image acquisitionunit. In this regard, the graphic may include a boundary surroundingboth of the eyes of the patient or boundaries surrounding each of theeyes of the patient, respectively. Also, the method may further includedisplaying information relating to at least one of: a diagnosed ocularmisalignment condition of the patient, a determined risk factor of thepatient developing a condition associated with ocular misalignment, andthe ocular misalignment measurements, on a display screen of the mobiledevice. The method may further include transmitting information relatingto at least one of: a diagnosed ocular misalignment condition of thepatient, a determined risk factor of the patient developing a conditionassociated with ocular misalignment, and the ocular misalignmentmeasurements, to an external vision care provider.

Moreover, the method may include associating information relating to atleast one of: a diagnosed ocular misalignment condition of the patient,a determined risk factor of the patient developing a conditionassociated with ocular misalignment, and the ocular misalignmentmeasurements, with personal information of the patient.

The method may further include acquiring, by the image acquisition unit,a video of the eyes of the patient, where the acquired image is a frameselected among a plurality of frames of the acquired video. In thisregard, the method may further include: acquiring, by the imageacquisition unit, the video of the eyes of the patient while a coversequence of the eyes of the patient is performed using an occluder; andgenerating, by the mobile device, one or more sounds to assist in theperformance of the cover sequence.

Furthermore, in accordance with embodiments of the present disclosure, amobile device includes: an image acquisition unit configured to acquirean image of eyes of a patient while light provided by a light sourcereflects from an optical surface of the eyes of the patient; and aprocessor configured to obtain ocular misalignment measurements,including a magnitude and a direction of ocular misalignment in the eyesof the patient, using the acquired image.

The mobile device may further include a flash producing device installedin the mobile device configured to produce a flash of light during theacquisition of the image, where the light source is the flash producingdevice.

Furthermore, in accordance with embodiments of the present disclosure, anon-transitory computer readable medium containing program instructionsfor performing a method includes: program instructions that obtainocular misalignment measurements, including a magnitude and a directionof ocular misalignment in eyes of a patient, using an image acquired byan image acquisition unit while light provided by a light sourcereflects from an optical surface of the eyes of the patient.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments herein may be better understood by referring to thefollowing description in conjunction with the accompanying drawings inwhich like reference numerals indicate identically or functionallysimilar elements, of which:

FIG. 1 is a diagrammatic view illustrating an exemplary mobile devicearchitecture;

FIG. 2 is a view showing an example of the mobile device acquiring animage of eyes of a patient;

FIG. 3 is a view showing an exemplary image of an eye of a patientacquired by the image acquisition unit;

FIGS. 4A and 4B are views illustrating an example of the mobile devicebeing used to perform heterophoria measurements;

FIG. 5 is a view of an example of the mobile device showing feedbackprovided to the user after processing of the acquired image;

FIGS. 6-8 are screen-views illustrating an exemplary mobile applicationuser interface;

FIGS. 9A and 9B are flowcharts showing an exemplary simplified procedurefor performing ocular misalignment measurement using the mobile deviceapplication described herein;

FIGS. 10A-10C are views of an exemplary video-based cover test mode;

FIG. 11 is a graph showing exemplary eye movement trace obtained whenperforming the video-based cover test shown in FIGS. 10A-10C;

FIG. 12 is a view showing an exemplary evaluation environment of themobile device application;

FIGS. 13A-13C are graphs showing exemplary data indicatingwithin-subject test-retest reliability of the deviation measurements bythe mobile device application;

FIG. 14 is a graph showing an exemplary comparison of measured deviationusing the mobile device application and the ground truth for differentangles of fixation; and

FIG. 15 is a graph showing exemplary results of phoria measurements forsubjects using the mobile device application.

It should be understood that the above-referenced drawings are notnecessarily to scale, presenting a somewhat simplified representation ofvarious preferred features illustrative of the basic principles of thedisclosure. The specific design features of the present disclosure,including, for example, specific dimensions, orientations, locations,and shapes, will be determined in part by the particular intendedapplication and use environment.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments of the present disclosure will be described indetail with reference to the accompanying drawings. As those skilled inthe art would realize, the described embodiments may be modified invarious different ways, all without departing from the spirit or scopeof the present disclosure. Further, throughout the specification, likereference numerals refer to like elements.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the disclosure.As used herein, the singular forms “a,” “an,” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof. As used herein, the term “and/or”includes any and all combinations of one or more of the associatedlisted items. The term “coupled” denotes a physical relationship betweentwo components whereby the components are either directly connected toone another or indirectly connected via one or more intermediarycomponents.

It is understood that the term “mobile device” or other similar term asused herein is inclusive of any portable computing device, such as smartphones, tablets, laptops, PDAs, and the like. A “mobile device,” as usedherein, is not necessarily limited to devices which are convenientlyportable, but may also include personal computers (PCs) or other similarcomputing machines. As referred to herein, a “mobile device” is equippedwith, at least, one or more processors, as is generally known in theart, and an image acquisition unit (e.g., camera) allowing for a user tocapture a photograph of a given subject. Further, a “mobile device” ispreferably equipped with communication components, either wired orwireless, allowing for the device to communicate with external devicesvia a communication network. Similarly, the terms “mobile deviceapplication,” “mobile application,” or “application,” as used herein,refer to a computer program executable by a processor installed in a“mobile device,” as is generally known in the art.

It is also understood that the term “patient” or other similar term asused herein is inclusive of any subject on whom an ocular misalignmentassessment could be performed. The term “user” as used herein isinclusive of any entity capable of interacting with or controlling amobile device. The “user” may also be the “patient,” or the “user” and“patient” may be separate entities, as described herein.

Additionally, it is understood that one or more of the below methods, oraspects thereof, may be executed by at least one processor. Theprocessor may be implemented in a mobile device, as described herein. Amemory configured to store program instructions may also be implementedin the mobile device, in which case the processor is specificallyprogrammed to execute the stored program instructions to perform one ormore processes which are described further below. Moreover, it isunderstood that the below methods may be executed by a mobile devicecomprising the processor, in conjunction with one or more additionalcomponents, as described in detail below.

Furthermore, the methods, or aspects thereof, of the present disclosuremay be embodied as non-transitory computer readable media on a computerreadable medium containing executable program instructions executed bythe processor. Examples of the computer readable mediums include, butare not limited to, ROM. RAM, compact disc (CD)-ROMs, magnetic tapes,floppy disks, flash drives, smart cards and optical data storagedevices. The computer readable recording medium can also be distributedin network coupled computer systems so that the computer readable mediais stored and executed in a distributed fashion, e.g., by a telematicsserver or a Controller Area Network (CAN).

Referring now to embodiments of the present disclosure, conditionsassociated with ocular misalignment, including amblyopia, strabismus,and heterophoria, among others, affect 3 to 5% of the United Statespopulation, with studies showing that only one-third of pre-schoolchildren in the United States receive any form of vision screening. Thecombined cost of amblyopia screening (using current technology) andstrabismus surgery in the United States is estimated to be $800 millionannually, while the cost to society from loss of function is estimatedat well over $20 billion. It is apparent, therefore, that accessibilityto convenient and inexpensive screening devices should be improved.

To this end, techniques are disclosed herein relating to a mobile deviceapplication for ocular misalignment measurement. Without using externalaccessories or attachments, the mobile application can measure ocularmisalignment, including, but not limited to, tropias (i.e., eyedeviation during binocular fixation) and phorias based on a snapshot ora series of snapshots of a patient's eyes captured by a camera equippedin the mobile device. After a photograph of the patient's eyes iscaptured, the application can automatically process the photograph tomeasure the magnitude of ocular misalignment. By taking advantage ofhigh-resolution cameras installed in many modern mobile devices, inconjunction with the custom designed image processing algorithmsdescribed herein, a high accuracy of measurement can be achieved. Theentire processing is performed locally on the mobile device itself;thus, the application is not required to send captured pictures to aremote server for processing. (Though, it is also possible to implementa cloud-based screening device whereby the captured pictures are sentfrom the mobile device to a remote server for processing.) Based on theobtained measurements, a condition (such as strabismus) can bediagnosed, and a risk factor for the patient developing a condition(such as amblyopia) can be quantified and provided to the patient on themobile device.

FIG. 1 illustrates an example diagrammatic view of a mobile devicearchitecture according to embodiments of the present disclosure. Asshown in FIG. 1, a mobile device 100 may contain multiple components,including, but not limited to, a processor (e.g., central processingunit (CPU) 110, a memory 120, a wireless communication unit 130, aninput unit 140, and an output unit 150. It should be noted that thearchitecture depicted in FIG. 1 is simplified and provided merely fordemonstration purposes. In view of the wide variety of commerciallyavailable mobile devices, the architecture of the mobile device 100,which is referenced throughout the present disclosure, can be modifiedin any suitable manner as would be understood by a person havingordinary skill in the art, in accordance with the present claims.Moreover, the components of the mobile device 100, themselves, may bemodified in any suitable manner as would be understood by a personhaving ordinary skill in the art, in accordance with the present claims.Therefore, the mobile device architecture depicted in FIG. 1 should betreated as exemplary only and should not be treated as limiting thescope of the present disclosure.

Components of the mobile device 100 will be briefly describedhereinbelow; though a detailed description thereof is well known in theart and thus will be omitted from the present disclosure. The processor110 is capable of controlling operation of the mobile device 100. Morespecifically, the processor 110 may be operable to control and interactwith multiple components installed in the mobile device 100, as shown inFIG. 1. For instance, the memory 120 can store program instructions thatare executable by the processor 110. The mobile application describedherein may be stored in the form of program instructions in the memory120 for execution by the processor 110. The wireless communication unit130 can allow the mobile device 100 to transmit data to and receive datafrom one or more external devices via a communication network. The inputunit 140 can enable the mobile device 100 to receive input of varioustypes, such as audio/visual input, user input, and the like. To thisend, the input unit 140 may be composed of multiple input devices foraccepting input of various types, including, for instance, a camera 142(i.e., an “image acquisition unit”), touch panel 144, microphone (notshown), one or more buttons or switches (not shown), one or more motionsensors (not shown), and so forth. The input devices included in theinput 140 may be manipulated by a user. For instance, a user can capturea photograph using the camera 142 by pressing the touch panel 144 in arecognized manner (i.e., a manner recognized by the processor 110). Thecamera 142 may include a front-facing camera and/or a rear-facingcamera. Notably, the term “image acquisition unit,” as used herein, mayrefer to the camera 142, but is not limited thereto. For instance, the“image acquisition unit” may refer to a program that acquires an imageof a patient stored locally in the memory 120 or remotely on a server.The output unit 150 can display information on the display screen 152for a user to view. The output unit 150 may further include a flashproducing device 154 (i.e., “flash”) which is a light source capable ofproducing a beam of light. The flash producing device 154 can beconfigured to produce a flash of light during acquisition of an image bythe camera 142.

As explained above, the mobile device 100 can be programmed in a mannerallowing it to perform the techniques for ocular misalignment screeningdescribed hereinbelow. In this regard, FIG. 2 illustrates an exampleview of the mobile device acquiring an image of eyes of a patient. Asshown in FIG. 2, the mobile device 100 can be positioned in front of theface of a patient 200 and aligned with the eyes of the patient 200 so asto acquire an image of the eyes of the patient 200 using an imageacquisition unit (e.g., rear- or front-facing camera 142) installed inthe mobile device 100.

While the user attempts to position the mobile device 100 to capture aphotograph of eyes of the patient 200, the mobile device 100 can provideguidance in real-time for the user in the form of instructions and/orgraphics 210 displayed on the display screen 152 to assist the user inproperly aligning the camera 142 with the patient 200, after which aphoto can be taken. For instance, the displayed feedback can guide theuser to align the mobile device 100 approximately 10 to 60 cm from theface of the patient 200, or more preferably approximately 20 to 50 cmfrom the face of the patient 200, or even more preferably approximately30 to 40 cm from the face of the patient 200, such that both eyes arepositioned within graphics 210 that are overlaid on the display screen152. The graphics 210 may include, for instance, a first boundary thatsurrounds both eyes of the patient 200 to indicate an area of interestand second boundaries that surround each detected eye of the patient200, respectively, as shown in FIG. 2. The displayed boundary orboundaries may be of a pre-defined shape, size, and/or position, or theshape, size, and/or position thereof may dynamically change based oninformation about the patient's eyes being detected by the camera 142and processed by the processor 110 in real-time.

The patient 200 can then be instructed by the user of the mobile device100 or by instructions displayed on the display screen 152 to fixate oneor both eyes on a point (e.g., the camera 142, a point on the mobiledevice 100 itself, an external object, etc.), thus allowing for properprocessing of the patient's eyes by the processor 110. In the case ofstrabismus testing, for example, measurement is performed binocularly,meaning that the patient 200 fixates on a specified target with botheyes, such as the camera 142.

The mobile device 100 can perform a photographic version of theHirschberg test for ocular misalignment measurements. To this end, alight provided by a light source reflects from a corneal surface of theeyes of the patient 200 during acquisition of the image of the eyes ofthe patient 200 by the camera 142. When light is shined in the eyes ofthe patient 200, its reflection can be seen on the cornea. Thus, thephotographic Hirschberg test, as implemented by the processor 110 of themobile device 100, can measure the decentration, i.e., the horizontaland/or vertical displacement, of the corneal reflection from the centerof the eye based on the acquired image of the patient's eyes, asdescribed in further detail with respect to FIG. 3. The light source maybe, for instance, the flash producing device 154 installed in the mobiledevice 100. As is known in the art, the flash producing device 154 canproduce a flash of light during acquisition of an image by the camera142. Alternatively, the light source may be any external light source(i.e., a light source independent of the mobile device 100, such as alamp, flashlight, etc.) capable of providing a light that reflects froma corneal surface of the eyes of the patient 200.

As the user attempts to align the mobile device 100 with the eyes of thepatient 200, such that a photograph sufficient for analysis by theprocessor 110 can be taken using the camera 142, the processor 110 cancompensate for a rotation of the mobile device 100 when the mobiledevice 100 is rotated. The compensation may be achieved in two ways.First, the processor 110 can determine the amount of tilt of the mobiledevice 100 with respect to the eyes of the patient 200 and the amount ofnecessary tilt compensation, e.g., using motion sensor(s) installed inthe mobile device 100, and real-time guidance can then be provided tothe user by displaying instructions and/or graphics on the displayscreen 152 indicating to the user that the mobile device 100 should betilted by an indicated amount. Second, the processor 110 can determinethe amount of tilt of the mobile device 100 with respect to the eyes ofthe patient 200 and the amount of necessary tilt compensation after animage of the patient 200 has been taken. In this case, the processor 110can perform image calculations to identify and compensate for the tiltpresent in the acquired image.

Once the user has aligned the mobile device 100 with the eyes of thepatient 200, a photograph of the patient's eyes can be captured using animage acquisition unit installed in the mobile device 100. As describedabove, the term “image acquisition unit,” as used herein, may refer tothe camera 142, but is not limited thereto. For instance, the “imageacquisition unit” may refer to a program that acquires an image of apatient stored locally in the memory 120 or remotely on a server. In thecase that the image acquisition unit is the camera 142 installed in themobile device 100, as shown in FIG. 2, the camera may be front-facing orrear-facing with respect to the mobile device 100. When using arear-facing camera, the patient 200 may require that a separate userperform the image acquisition process (e.g., by holding the mobiledevice 100, aligning the mobile device with the patient's eyes, andpressing the designated shutter button as recognized by the processor110), which is specifically shown in FIG. 2. When using a front-facingcamera, however, the patient 200 may perform the image acquisitionprocess by himself or herself, as the patient 200 is capable of viewingthe display screen 152 to properly align the camera 142 with thepatient's eyes prior to capturing the photograph.

The magnitude of ocular misalignment in the eyes of the patient 200 canbe measured using a variety of techniques, each of which requiringdetecting one or more structures or features of the eyes of the patient200. In this regard, FIG. 3 illustrates an example image of an eye of apatient acquired by the image acquisition unit. As shown in FIG. 3, theimage 300 includes multiple features that can be detected by theprocessor 110 during processing of the image 300, including the eyecenter 310, the corneal reflection 320, the limbus boundary 330, theiridoscleral border, the pupillary margin, and the like, for the purposeof measuring ocular misalignment, including the amount and direction ofmisalignment.

Detecting such structures or features in the acquired image 300 may beachieved using a variety of suitable image processing algorithms, suchas an adaptive thresholding algorithm, for example. According to onetechnique for measuring ocular misalignment, which is particularlyuseful for detecting the presence of strabismus in the patient 200, theprocessor 110 can detect in the acquired image 300 of the patient 200 anocular reflection 320 on the corneal surface of both eyes and comparethe position of the ocular reflection 320 in each eye to one another.More specifically, the processor 110 can compute a distance d between areference point 310 (e.g., the eye center/pupil or iris) in each of theleft and right eyes and the corneal reflection 320 caused by the light.The eye center can be calculated by computing a curve fitted to thelimbus boundary 330. The position of the reference point 310 would bethe same in each eye, but the position of the corneal reflection 320 ineach eye may vary based on the amount of misalignment present in theeyes of the patient 200. Notably, the processor 110 can detect featuresin the eyes of the patient 200 regardless of the patient's eye color.Along these lines, the processor 110 can use the pupil region as areference point for eyes with light color, rather than using the iris asa reference point.

When the eyes are aligned with each other, the distance d between thereference point 310 of the eye and the corneal reflection 320 isapproximately the same in both the eyes, or less than a predeterminedmisalignment threshold. The misalignment threshold may be set by theclinician or user (e.g., one prism diopter (PD), two PD, three PD, etc.)based on a desired sensitivity and/or may be calibrated according toaccuracy of the system being operated by processor 110. For instance, apresence of excessive noise in the system creates the risk forinaccurate measurements, and the misalignment threshold can be increasedaccordingly.

Conversely, a difference in the distance d between the two eyes isindicative of misalignment (any manifest eye deviation present underbinocular fixation is termed as a tropia). To this point, a differencecan be calculated between the distance d from the reference point 310 inthe left eye to the corneal reflection 320 in the left eye (i.e., “firstdistance”) and the distance d from the reference point 310 in the righteye to the corneal reflection 320 in the right eye (i.e., “seconddistance”). The calculated difference may then be compared to apredetermined misalignment threshold to ascertain whether the differencebetween the first distance and the second distance (i.e., themisalignment of the left and right eyes) is significant enough todiagnose a condition of the patient (e.g., strabismus, heterophoria,etc.). If the calculated difference is greater than or equal to thepredetermined misalignment threshold, the processor 110 may determinethat the patient 200 is suffering from a condition associated withocular misalignment. The predetermined misalignment threshold may bestored in the memory 120 and retrieved from the memory 120 by theprocessor 110. The misalignment threshold may be predefined to a defaultvalue, such as three prism diopters, for example, and may be tunedaccording to the preference of the user.

In order to calculate the distance d between the reference point 310 andthe ocular reflection 320 in a given eye, the position of the referencepoint 310 and the position of the ocular reflection 320 can be measuredin image space using the acquired image 300 of the eyes of the patient200. To this end, the positions of the reference point 310 (e.g., iris,pupil, limbus boundary, or the like) and the ocular reflection 320 isexpressed in image space coordinates (e.g., pixels). In this manner, thedistance d between the reference point 310 and the ocular reflection 320in a given eye can be readily calculated.

Furthermore, the calculated distance d between the reference point 310and the ocular reflection 320 can be converted into an objective measuresuch as prism diopters or degrees using a Hirschberg ratio (HR) computedfor the patient 200 based on the image 300 and an internal calibrationfactor based on iris diameter. The Hirschberg ratio describes therelationship between the corneal reflection 320 decentration and thedeviation of the eye. That is, the Hirschberg ratio determines how muchangular deviation an eye would undergo for each millimeter ofdecentration of the corneal reflection 320. This ratio can be determinedusing population averages and can vary by about 4.5 prism diopters(standard deviation) from the mean value for a given patient. Thisvariation ultimately affects the deviation measurement.

However, an accurate Hirschberg ratio for an individual can bedetermined if the patient 200 binocularly fixates on targets placed atknown eccentricities with respect to the camera 142 (i.e., non-primarygaze directions, where the head is not facing forward). A plurality ofimages corresponding to each fixation, respectively, can be captured bythe camera 142, and the eye deviations (both eyes separately) can beobtained in each image 300 using the techniques described above.Guidance can be provided to the user, e.g., by instructions and/orgraphics displayed on the display screen 152, to assist the user inpositioning the mobile device 100 to obtain a predefined angular changein position of the different targets (either on the mobile device 100itself or a point that is not on the mobile device 100) relative to theeyes during the acquisition of the plurality of images. That is, theuser can be assisted to obtain a predefined angular extent. Therelationship between actual and measured eye deviations may then be usedto determine the individualized Hirschberg ratio. That is, theHirschberg ratio for the patient 200 call be calculated based on themeasured magnitudes of ocular misalignment in the acquired images 300.Notably, the individualized Hirschberg ratio depends on the iris, pupilposition, and angle Kappa of the patient 200. These, too, can bemeasured by the processor 110 when processing the acquired image 300.

In addition to performing Hirschberg ratio calculations, a plurality ofimages 300 of the patient 200 can be acquired in order to enhance thequality of the image to be processed by the processor 110. In oneexample, after acquiring a plurality of images 300 of the patient 200,the processor 110 can examine the images 300 to select one of the imageshaving the highest quality. Then, the processor 110 can process thehighest quality image 300 for the measuring of the magnitude of ocularmisalignment. In another example, after acquiring a plurality of images300 of the patient 200, ocular misalignment measurements can be obtainedin each of the acquired images 300, and a composite of the obtainedocular misalignment measurements is generated. Then, an overall ocularmeasurement can be determined based on the generated composite.Additionally, measurements of deviation could be performed in eachcaptured image, and an average or median of the measurements can betaken.

While tropias are manifest eye deviations, heterophorias (or phorias)are latent deviations which are not observed under binocular viewingconditions. As explained above, heterophorias occur when the fusion ofthe two eyes is broken with only one eye fixating. The non-fixating ordeviating eye deviates from its initial position when not being used andreturns to its normal position when the fusion is established again.

Thus, in order to measure heterophoria, another technique for measuringocular misalignment can be performed, in which the patient 200 fixateson a target that is visible only to one eye as the mobile device 100, orother vision occluding object, blocks the other eye from viewing thetarget. This way, the fusion between the two eyes is broken, and theprocessor 110 can measure the deviation between the viewing eye andblocked eye.

In this regard, FIGS. 4A and 4B illustrate example views of the mobiledevice being used to perform heterophoria measurements. As shown in FIG.4A, a user may hold the mobile device 100 to acquire an image of theeyes of the patient 200, as previously demonstrated in FIG. 2. Here,however, the patient 200 fixates on an external target (or stimulus) 410while the mobile device 100 is positioned such that only one of thepatient's eyes is able to view the stimulus 410. As shown in FIG. 4B,the perspective of the patient (or subject) 200 shows that one eye has aclear, unblocked view of the external stimulus 410, while the other eyeis precluded from viewing the stimulus by the positioning of the mobiledevice, allowing the blocked eye to deviate. Furthermore, whenperforming heterophoria measurements, the image acquisition unit maycapture a plurality of images of the patient 200 fixating at near andfar distances (using only one eye). The captured images can then beanalyzed by the processor 110 to compute the amount of deviation betweenthe eyes.

Notably, the results of a heterophoria measurement can be compared tothe results of a strabismus measurement to diagnose and/or quantifyeither strabismus or heterophoria. When no misalignment is detected inthe strabismus test, but misalignment is detected in the heterophoriatest, a heterophoria can be diagnosed. On the other hand, whenmisalignment is detected in the strabismus test, the diagnosis isstrabismus and not heterophoria, regardless of the results of theheterophoria test.

In addition, the Kappa angle can be calculated by the processor 110based on the acquired image 300 of the patient 200 to assist inmeasuring the magnitude of ocular misalignment. The Kappa angle is theangle between the optical and the visual axis of the eye. Because thefovea is not completely aligned with the optical axis of the eye, theremay be some offset that varies between individuals; the Kappa anglequantifies this offset.

The Kappa angle can be computed by the processor 110 when a single eye(monocular) of the patient 200 fixates on a light source (e.g., flashproducing device 154), with the other eye covered. In this position, thecamera 142 can acquire an image 300 of the patient 200, such that theprocessor 110 detects and processes the corneal light reflectiondecentration in one eye at a time, rather than performing differentialmeasurements. Then, once the iris diameter and Hirschberg ratio arecalculated, using the techniques described above, the Kappa angle can becalculated in terms of degrees by the processor 110. The calculatedKappa angle can further assist in diagnosing a condition of the patient200. Also, the processor 110 can diagnose which eye of the patient 200is misaligned and a misalignment direction of the misaligned eye basedon the Kappa angle. Along these lines, the processor 110 can measure afixation reliability index of the non-misaligned eye based on adeviation of the non-misaligned eye from the Kappa angle.

In one example, the Kappa angle can be used to measure whethermisalignment is occurring in the right or left eye, as well as thedirection of misalignment (e.g., eso or exo). Here, the processor 110can select the eye with the Kappa angle closer to the predeterminedpopulation norm as the fixating eye (i.e., non-misaligned eye). Theprocessor 110 can then measure the fixation reliability index, asmentioned above, based on the deviation of the fixating eye from thepopulation norm Kappa angle. These measurements can be displayed on thedisplay screen 152 for the patient's reference in the form of a score, agraphic, or other similar technique.

In another example, the processor 110 can measure the horizontal irisdiameter of the patient 200 and use the diameter as the reference point.The processing algorithm can then be calibrated using a predeterminedpopulation norm of the reference point, for example, a horizontal irisdiameter (HID) of 11.7 mm, or a physical measurement of the irisdiameter performed by the patient 200 entered into the mobile device100. The calibrated measurements can be used to convert biometricmeasurements (e.g., interpupillary distance, pupil diameter, Kappaangle, etc.) made in pixel space to physical space. As a result, theprocessor 110 can determine the consistency of testing distance from thepatient's eye to the mobile device 100 on successive measurements usingchanges in the horizontal iris diameter or other biometric landmarkbetween tests.

Moreover, the processor 110 may calculate an accommodativeconvergence/accommodation (AC/A) ratio (or convergenceaccommodation/convergence (CA/C) ratio) of the eyes of the patient 200based on the acquired image 300 and the measured magnitude of ocularmisalignment. A measurement of heterophoria in the eyes of the patient200 can also assist in the calculation of the AC/A ratio. The AC/A ratiois a measure of how the two eyes work together, that is, how well theeyes can fixate on a single target. To attain the AC/A ratio, the ocularmisalignment of the patient 200 may be measured in multiple iterationsunder different conditions, where the patient 200 attempts fixates withboth eyes on the same object. For example, a first photograph of thepatient 200 can be taken at a distance of one meter or greater away fromthe camera 142, where there is essentially no convergence/accommodationdemand, and a second photograph of the patient 200 can be taken at ashorter distance, such as 40 centimeters. Using these multiple sets ofmeasurements, the processor 110 can compute the AC/A ratio of thepatient 200.

After the acquired image 300 of the patient 200 has been processed bythe processor 110 to measure a magnitude of ocular misalignment, asexplained in detail above, feedback can be provided to inform thepatient 200 of the measurements taken. In this regard, FIG. 5illustrates example feedback provided to the user (or patient) afterprocessing of the acquired image. As shown in FIG. 5, after capture ofthe image 300 and subsequent processing of the image 300, feedback 510indicating results of the measurements performed by the processor 110can be displayed on the display screen 152 for the user's (or patient's)reference. The feedback 510 may include any information relevant to thecomputations performed by the processor 110 in processing the acquiredimage 300 of the patient 200 including, for example, a position of thepupil, a position of the iris, a position of the corneal reflection, eyedeviation values, Hirschberg ratio, Kappa angle, AC/A ratio, and thelike.

Further, the patient 200 may be informed of information indicating adiagnosed condition, such as strabismus, heterophoria, etc., of thepatient 200 and/or a risk factor of the patient 200 developing acondition associated with ocular misalignment, such as amblyopia. Thediagnosis and risk can be determined by the processor 110 based on theobtained ocular misalignment measurements, e.g., the amount ofmisalignment in each eye, the magnitude and/or direction ofmisalignment, etc.

The techniques disclosed herein can be applied for detecting relatedocular conditions, such as nystagmus. Nystagmus is characterized byintermittent eye misalignment motions in the horizontal or verticalaxis. The camera 142 can acquire a serial image or video along with acontinuous or flashing light produced by the flash 154. Notably,nystagmus movements above a certain frequency of occurrence within agiven time frame and eccentricities have been positively correlated withblood alcohol levels and are one of the roadside tests commonly used bypolice officers. Thus, the processor 110 may measure and interpret thefrequency of events of misalignment occurring above a certain thresholdmagnitude and within a defined time/space span.

The mobile device 100 is capable of transmitting information relating tothe ocular misalignment measurements, a diagnosis of the patient 200,and/or the determined risk of the patient 200 developing a conditionassociated with ocular misalignment to an external vision care provider(e.g., using the wireless transmission unit 130). The information can betransmitted automatically or in response to user input. Also, theinformation can be sent to the care provider via any suitablecommunication medium, including, but not limited to, wirelesstransmission, wired transmission, physical transfer (e.g., USB drive,paper records, etc.), and the like. This way, an external vision careprovider can become aware of patients who may be in need of treatmentand can develop treatment plans for such patients more efficiently.

FIGS. 6-8 illustrate example screen-views of a mobile application userinterface. It is understood that the user interfaces depicted in FIGS.6-8 do not limit the scope of the present claims and are merelypresented for demonstration purposes. Thus, the user interface of themobile application described herein may be modified in any suitablemanner, as would be understood by a person of ordinary skill in the art.

As shown in FIG. 6, a registration page 600 can be presented to thepatient 200. Here, the patient 200 can enter preliminary personalinformation. The processor 110 can associate the personal information ofthe patient 200 with the ocular misalignment measurements, the diagnosedcondition of the patient 200, and/or the determined risk factor of thepatient 200 developing a condition associated with ocular misalignment.The associated personal information may also be transmitted to theexternal vision care provider. The personal information may be storedlocally in the memory 120 or remotely on a server. Alternatively, thepersonal information may be discarded. Storage of the above informationmay be controlled by the patient 200. For confidentiality purposes, theapplication may prompt the patient 200 for consent to store, send, orotherwise use the personal information of the patient 200.

As shown in FIG. 7, a profile page 700 can be presented to the patient200 or to another user, such as a friend or family member of the patient200, a medical professional, and so forth. Here, the profile page 700can provide the patient's personal information, as well as a previouslyacquired photo(s) of the patient 200, if available. Additionally, if amisalignment assessment for the patient 200 was previously performed,the profile page 700 can provide information indicating the ocularmisalignment measurements, the diagnosed condition, and/or thedetermined risk factor of the patient 200 developing an ocularmisalignment condition. The information may be retrieved from the memory120 or from a remote server.

As shown in FIG. 8, an image acquisition page 800 can be presented tothe patient 200 allowing a user, or the patient himself or herself, tocapture a photograph of the eyes of the patient 200. As explained withrespect to FIG. 2, one or more boundaries 210 may be displayed on theimage acquisition page 800 to assist the user in aligning the mobiledevice 100 with the patient's eyes. Additional controllable on-screenobjects may be displayed on the image acquisition page 800, including,for instance, brightness control, flash activation, front- orrear-facing camera activation, zoom control, shutter button, and thelike. After an image 300 of the patient 200 has been captured by thecamera 142 (or otherwise acquired), and the acquired image 300 of thepatient 200 has been processed by the processor 110 to measure amagnitude of ocular misalignment, feedback 510 can be provided to informthe patient 200 of the measurements taken, as demonstrated in FIG. 5.

FIGS. 9A and 9B illustrate an example simplified procedure forperforming ocular misalignment measurement using the mobile deviceapplication described herein. The procedure 900 may start at step 905,and continue to step 910, where, as described in greater detail herein,an image of the eyes of a patient 200 can be acquired and processed bythe processor 110 to determine a magnitude of ocular misalignment in thepatient's eyes.

At step 910, detection boundaries 210 can be displayed on a camerapreview screen, such as the image acquisition page 800 shown in FIG. 8.The camera preview screen can be presented to the user on the displayscreen 152 of the mobile device 100 and can include a preview image ofthe subjects currently in view through the lens of the camera 142.

At step 915, the processor 110 can determine whether the mobile device100 is properly aligned with the eyes of the patient 200. As the userattempts to align the mobile device 100 with the eyes of the patient200, such that a photograph sufficient for analysis by the processor 110can be taken using the camera 142, the processor 110 can automaticallycompensate for a rotation of the mobile device 100 when the mobiledevice 100 is rotated, such that the camera plane is parallel to theface of the patient 200. The compensation may be achieved in two ways.First, the processor 110 can determine the amount of tilt of the mobiledevice 100 with respect to the eyes of the patient 200 and the amount ofnecessary tilt compensation, and real-time guidance can be provided tothe user by displaying instructions and/or graphics on the displayscreen 152 indicating to the user that the mobile device 100 should betilted by an indicated amount (step 925). Second, the processor 110 candetermine the amount of tilt of the mobile device 100 with respect tothe eyes of the patient 200 and the amount of necessary tiltcompensation after an image of the patient 200 has been taken. In thiscase, the processor 110 can perform calculations to identify andcompensate for the tilt present in the acquired image.

Once the user has properly aligned the mobile device 100 with thepatient 200, the focus of the camera 142 can be adjusted—eitherautomatically or manually by the user (e.g., using on-screencontrols)—so the patient's eyes are in clear view (step 930).

At step 935, the processor 110 can determine whether light is reflectingfrom a corneal surface of the eyes of the patient 200. The user caninform the processor 110 that an external light is being used, or theprocessor 110 can automatically detect if sufficient light is reflectingfrom the corneal surface of the patient's eyes.

If it is determined that an external light source is not being used, atstep 940, the flash producing device 154 can be activated. As is knownin the art, the flash producing device 154 can produce a flash of lightduring acquisition of an image by the camera 142. Thus, a photographicHirschberg test, as implemented by the processor 110 of the mobiledevice 100, can measure the decentration, i.e., the horizontal and/orvertical displacement, of the corneal reflection from the center of theeye based on the acquired image of the patient's eyes, as described infurther detail below.

At step 945, the camera 142 can capture an image 300 of the patient 200.As described above, the term “image acquisition unit,” as used herein,may refer to the camera 142, but is not limited thereto. For instance,the “image acquisition unit” may refer to a program that acquires animage of a patient stored locally in the memory 120 or remotely on aserver. In the case that the image acquisition unit is the camera 142installed in the mobile device 100, the camera may be front-facing orrear-facing with respect to the mobile device 100. When using arear-facing camera, the patient 200 may require that a separate userperform the image acquisition process, which is specifically shown inFIG. 2. When using a front-facing camera, however, the patient 200 mayperform the image acquisition process by himself or herself, as thepatient 200 is capable of viewing the display screen 152 to properlyalign the camera 142 with the patient's eyes prior to capturing thephotograph.

At step 950, the acquired image 300 of the patient 200 can be processedby the processor 110 to detect a plurality of features in the eyes ofthe patient 200. The magnitude of ocular misalignment in the eyes of thepatient 200 can be measured based on the detected features. As shown inFIG. 3, the acquired image 300 includes multiple features that can bedetected by the processor 110 during processing of the image 300,including the eye center 310, the corneal reflection 320, the limbusboundary 330, the iridoscleral border, the pupillary margin, and thelike, for the purpose of measuring ocular misalignment.

At step 955, the detected features can be displayed to the patient 200.The display screen 152 can provide the patient 200 with text-basedinformation indicating the positions of the detected features listedabove and/or the acquired image 300 of the patient 200 with graphicsoverlaid on the image 300 indicating the positions of the detectedfeatures.

At step 960, the patient 200 can confirm whether the results areacceptable. For instance, if the image 300 is blurry or tilted or thepatient's eyes are not clearly visible, positions of the detected eyefeatures may be erroneous. In such case, the patient 200 can opt tocapture a new photograph, causing the procedure to return to step 915.Alternatively, the processor 110 can inform the patient 200 that theimage 300 will not be sufficient for obtaining accurate measurements andprompt the user to capture a new photograph of the patient 200.

At step 965, the processor 110 measures the ocular misalignment based onthe detected features of the patient's eyes in the acquired image 300.For instance, according to one technique for measuring ocularmisalignment, which is particularly useful for detecting the presence ofstrabismus in the patient 200, the processor 110 can detect in theacquired image 300 of the patient 200 an ocular reflection 320 on thecorneal surface of both eyes and compare the position of the ocularreflection 320 in each eye to one another. More specifically, theprocessor 110 can compute a distance d between a reference point 310(e.g., the eye center) in each of the left and right eyes and thecorneal reflection 320 caused by the light. The eye center can becalculated by computing a curve fitted to the limbus boundary 330. Theposition of the reference point 310 would be the same in each eye, butthe position of the corneal reflection 320 in each eye may vary based onthe amount of misalignment present in the eyes of the patient 200. Then,a difference can be calculated between the distance d from the referencepoint 310 in the left eye to the corneal reflection 320 in the left eye(i.e., “first distance”) and the distance d from the reference point 310in the right eye to the corneal reflection 320 in the right eye (i.e.,“second distance”).

At step 970, the calculated measurements can be converted into anobjective measure such as prism diopters or degrees. The conversion canbe performed using a Hirschberg ratio for the patient 200, whichdescribes the relationship between the corneal reflection 320decentration and the deviation of the eye, and an internal calibrationfactor based on iris diameter. Calculations for the Hirschberg ratio aredescribed in detail above.

At step 975, the measurements of ocular misalignment can be presented tothe patient 200. As shown in FIG. 5, feedback 510 indicating results ofthe measurements performed by the processor 110 can be displayed on thedisplay screen 152 for the patient's reference. The feedback 510 mayinclude any information relevant to the computations performed by theprocessor 110 in processing the acquired image 300 of the patient 200including, for example, a position of the pupil, a position of the iris,a position of the corneal reflection, eye deviation values, Hirschbergratio, Kappa angle, AC/A ratio, and the like. The measurements can alsobe stored in memory, either locally in memory 120 or remotely on aserver.

At step 980, the processor 110 can determine whether the measurementsare greater than or equal to a threshold value. In particular, thecalculated difference between the “first distance” and the “seconddistance” may be compared to a predetermined misalignment threshold toascertain whether the difference between the first distance and thesecond distance (i.e., the misalignment of the left and right eyes) issignificant enough to determine that the patient is suffering from acondition associated with ocular misalignment (e.g., strabismus,heterophoria, etc.). If the calculated difference is greater than orequal to the predetermined misalignment threshold, the processor 110 maydiagnose the patient 200 as having an ocular misalignment condition. Thepredetermined misalignment threshold may be stored in the memory 120 andretrieved from the memory 120 by the processor 110. The misalignmentthreshold may be predefined to a default value, such as three prismdiopters, for example, and may be tuned according to the preference ofthe user.

At step 985, the processor 110 can determine whether the patient 200 hasa condition associated with ocular misalignment, such as strabismus,based on the obtained ocular misalignment measurements. The processor110 can determine a confidence level (i.e., a likelihood) of the patient200 having strabismus, according to the measurements performed in steps965 and 970. The display screen 152 can display these results to thepatient 200. Additionally, the results can optionally be transmitted toan external vision care provider (e.g., using the wireless transmissionunit 130).

The procedure 900 illustratively ends at step 990. The techniques bywhich the steps of procedure 900 may be performed, as well as ancillaryprocedures and parameters, are described in detail above.

It should be noted that the steps shown in FIG. 9 are merely examplesfor illustration, and certain other steps may be included or excluded asdesired. Further, while a particular order of the steps is shown, thisordering is merely illustrative, and any suitable arrangement of thesteps may be utilized without departing from the scope of theembodiments herein. Even further, the illustrated steps may be modifiedin any suitable manner in accordance with the scope of the presentclaims.

Example 1: Evaluation of Mobile Device Application for Measuring EyeAlignment

The mobile device application described herein performs an automatedphotographic Hirschberg test with goals of obtaining rapid, convenient,and quantifiable measurements of both manifest and latent eyedeviations. To use the application, an examiner may ask the patient tofixate on a point (e.g., a rear-facing camera of the mobile device), andeither capture a photo or record a video while covering or uncoveringthe patient's eyes. The mobile device application processes the capturedimage or video and computes the eye deviation based on de-centration ofthe corneal reflection.

In an effort to evaluate the mobile device application, eye deviationsmeasured by the application were compared to the ground truth and withinsubjects in two separate trials for 25 normally sighted subjectsfixating monocularly on targets covering an angular range of 464 (prismdiopters (PD) can be shown using the symbol A). In another experiment,phoria measurements of 14 normally sighted subjects were comparedbetween the application and the Modified Thorington (MT) method.

The results, as described below, demonstrate that a 95% confidenceinterval for repeated Hirschberg ratios (HR) obtained by the mobiledevice application was ±1.57Δ. Eye deviations measured by theapplication were correlated with the ground truth (R²=1.003; p<0.001),with a root mean squared error (RMSE) of 1.7Δ for eye deviations within±10Δ (2.4Δ over entire range). Phoria measurements with the applicationwere strongly correlated with MT (R²=0.97; p<0.001), with an error of−0.76±1.6Δ. Accordingly, the mobile device application described hereincan accurately and reliably measure a wide range of eye deviations andphoria that correlate closely with the conventional clinical method ofMT.

A. MOBILE DEVICE APPLICATION

Using only a mobile device equipped with a camera, the mobile deviceapplication described herein can automatically detect ocularmisalignment and provide an objective measure of eye deviation eitherfrom a single captured image or from a series of images of the patient'seyes. The patient may be asked to look at the mobile device camera, orany other suitable fixation target, depending on the test to beperformed. Corneal reflection can be generated using either a built-inflash of the mobile device or external light source, such as lamp or eyechart light box. Processing of acquired images can be performed entirelywithin the mobile device's operating system, without the need for aninternet connection or a remote server.

Strabismus measurements are typically performed with the patientfixating with both eyes on a specified target (e.g., a rear-facingcamera of the mobile device). Image processing algorithms describedherein can detect relevant features within each eye, including thelimbus boundary, eye center, and corneal reflection, as shown in FIG. 3.The distance between the center of the eye and the corneal reflectionmay be computed, where the difference in computed distance between thetwo eyes may be converted to an objective measure using a HR in therange of previously reported values and an internal calibration factor,as described herein.

Under manual inspection, the Hirschberg test is known to have highvariability in the outcome and is meant for obtaining a gross estimateof the deviation. However, a precise measurement can be achieved byemploying high-resolution mobile device cameras and specialized imageprocessing algorithms. Since the internal calibration factor, which isthe average value of human horizontal visible iris diameter (HVID), isrelatively constant throughout human populations, it serves as arelatively robust automatic calibration factor for the mobile devicecamera. Consequently, the mobile device can be utilized in the mannerdescribed herein without having to explicitly measure the HVID for eachsubject and without major constraints on the positioning of the mobiledevice camera from the eyes to be imaged. Data presented belowdemonstrate that use of the population average value of HVID instead ofindividual values does not substantially affect the accuracy of eyedeviation computation. Eye deviation can be measured monocularly withrespect to a ground truth, or binocularly (e.g., see FIG. 2) between thetwo eyes with the subject fixating on a suitable target.

B. VIDEO-BASED COVER TEST MODE

The mobile device application is capable of measuring eye deviationsunder dissociated conditions. In certain situations, such as in the caseof intermittent strabismus or phoria, the eye deviation is not presentwhen fixating binocularly. The eye deviation can be seen only whenbinocular fusion is broken (e.g., dissociated condition). In a clinic,such deviations can be measured using a cover test with prismneutralization (CTPN), where an occluder is placed in front of an eye toperform either cover-uncover or alternate-cover tests with prisms. Thisfunctionality is adopted in the mobile device application describedherein for measurement of latent eye deviations using an automated covertest without using prisms.

To perform a “video-based cover test,” the examiner can place the mobiledevice approximately 10 to 50 cm, or more preferably approximately 20 to40 cm, or even more preferably approximately 30 cm from the patient'seye and press a button to record a video (one or more serial photos canbe captured in other modes, as described above), while the patientfixates on a target binocularly. The patient's binocular fusion isbroken by performing a cover sequence of the patient's eyes using anoccluder 1000. e.g., cover-uncover (of one eye) or alternate cover(involving both eyes), as demonstrated in FIGS. 10A-10C. As shown inFIGS. 10A and 10B, the eyes of the patient are alternately covered usingthe occluder 1000, and as shown in FIG. 10C, the eyes of the patient(including the non-dominant eye) are ultimately uncovered. The mobiledevice 100 can record the entire cover sequence event as a video.

The video frame just after uncovering the non-dominant eye can beselected for processing the eye deviation. Due to the high frame-rate ofvideo capture (e.g., 30 frames per second), the deviation can bemeasured just as the eye is uncovered and before it can recover and gainfixation. FIG. 11 shows an example of the eye movement trace obtainedwhen performing the video-based cover test shown in FIGS. 10A-10C, withthe mobile device 100 recording the video of this event (i.e., “coversequence”). The movements of eyes can be seen as they are alternatinglyuncovered, and for a brief duration after removing the cover (e.g.,occluder 1000), the eye deviation (A) can be measured before the eyemoves back into its fixating position. There is a brief duration afterremoving the cover during which the deviation can be measured (denotedby the shaded zone 1100). To aid in ease of use and consistency of thevideo cover test performance, the mobile device application can provideaudio tones to guide the patient in covering/uncovering the eyes duringthe cover sequence.

C. EVALUATION OF MOBILE DEVICE APPLICATION

The accuracy and reliability of the mobile device application describedherein in measuring the absolute angle of deviation was evaluated in acarefully calibrated laboratory setup. As shown in FIG. 12, subjects 200rested their heads in a chin-rest and faced the rear side of a mobiledevice 100 that was placed at approximately 30 cm from their eyes. Theheight of the apparatus was adjusted as needed. The distance of themobile device's built-in camera 142 to the eyes of the subject 200 wasmeasured manually using a ruler. Fixation targets (black markings 1200on white background) were pasted on the back surface of the mobiledevice 100, as shown in FIG. 12, such that they aligned with the camera142. The reason for such a placement was to ensure symmetriceccentricities of the fixation targets with respect to an obviousreference point. There were 13 targets 1200, placed 1 cm apart andcovering an angular range of 26° (about 46Δ) overall to measure eyedeviations along the horizontal direction. The subjects were asked tofixate on the targets 1200 only using their right eye while the left eyewas patched. This was done to enhance the fixation stability and reduceany possible causes of error due to any other latent binocular visionlimitations of the subjects. Measurements were taken for each fixationposition in two separate trials so as to determine the test-retestaccuracy of the mobile device application.

The application captured an image with a flash at each fixation point1200. After the application processed the captured image for a givenfixation point 1200 (within a fraction of seconds), the examinerinstructed the subject to move the gaze to the next point 1200. In casethe application failed to record (e.g., due to the subject blinking),measurement was repeated for the current fixation target 1200. The nexttrial was conducted in the same session after a short break of fiveminutes.

25 normally sighted subjects without any known strabismus participatedin the eye deviation measurement experiment. The subjects wererelatively young (between the ages of 20 and 40), and their correctedvisual acuity was no worse than 20/30. This group also included thosewho habitually wore prescription eye glasses; however, they did not wearthem during the experiment to avoid issues due to specular reflectionsinterfering with the operation of the application. Only those subjectswho could resolve the fixation targets 1 cm apart at 30 cm withoutwearing their glasses were included in the study.

In a second experiment, phoria measurement was performed using thevideo-based alternate cover test mode described above in 14 normallysighted non-strabismic young adults (age less than 40). Similar to theearlier setup, the subject 200 rested his or her head in a chin rest andthe phone was placed at about 40 cm from the eyes. An accommodativetarget served as a fixation point which was attached near the mobiledevice's camera 142. The eyes were alternatingly covered and uncoveredby the examiner with an occluder 1000 while the subject maintainedfixation. A short video of about two seconds was captured during thecover sequence. Auditory tones generated by the mobile device 100 guidedthe examiner to change the cover from one eye to another and finallycompletely remove the cover. The audible tones were timed such that itrecorded the last two seconds of cover sequence that included uncoveringof both the eyes. Three frames just after uncovering the eyes wereselected for processing, and the maximum magnitude of deviation computedin these three frames was recorded as the output of the application.Since the video capture was at high-speed, sometimes the frame justafter removal of the occluder 1000 was blurry. To alleviate theinaccuracies in measurement due to the blur, the strategy of usingmaximum deviation in three consecutive frames after uncovering wasemployed. Each subject was tested three times with the mobile deviceapplication. For comparison, phoria was measured using ModifiedThorington (MT) near card (i.e., at 40 cm) twice, and repeated once moreif the difference between the first two measurement exceeded 2Δ.

D. STATISTICAL ANALYSIS

The eye deviation measurement results of the mobile device applicationdescribed herein were manually examined for each fixation point 1200 andany erroneous detections were discarded (e.g., incorrect cornealreflection localization or incorrect limbus boundary estimation). Theapplication computed eye deviation in mm (the pixel level decentrationwas scaled by the average iris diameter). The amount of angulardeviation an eye would undergo for each mm of corneal reflectionde-centration is determined by the HR, which varies from person toperson. The slope of the line fitting the scatter of the ground truthangular deviations versus the mm deviations of the eye measured by theapp gives the estimated HR for an individual, while the intercept isindicative of the kappa angle (i.e., the angle between the visual axisand the optical axis). The test-retest reliability of the application inestimating the HR and kappa angle for each subject was also measured. ABland-Altman or Tukey mean difference plot with 95% confidence interval(CI) of the distribution was used to visualize these results, as shownin FIGS. 13A-13C. Furthermore, in order to determine the accuracy of theapplication in estimating absolute eye deviation angles, themeasurements for each individual around the central fixation point werenormalized and scaled using the population averaged HR. The average HRused for a subject 200 was calculated from all the remaining subjects(excluding the data from that particular subject). Ground truth measuresof the HVID were factored in to reconcile a potential source ofbetween-subject variability.

E. RESULTS

With two trials performed for each subject, and 13 fixation points 1200per trial, the mobile device application processed 26 images per subjectand 650 images in total. Out of these, the application was able tocorrectly process 629 images.

FIGS. 13A-13C shows the within-subject test-retest reliability of thedeviation measurements by the application for the slope (HR), intercept(angle-kappa), and the R² values for lines fitting the measured and theground truth eye deviation data. The Bland-Altman plots shown in FIGS.13A-13C include dash and dot lines representing the 95% CI of agreementbetween two measurements. The solid line represents the mean differencebetween two measurements over the entire population and the dashed linesare its 95% CI limits.

In detail, the average ±standard estimated HR, kappa angle, and R² overthe study population were 19.26±1.78 Δ; 5.49±3.48Δ (nasal); and0.992±0.0026, respectively. The average difference in HR, kappa angle,and R² values estimated in two trials were 0.13±0.31 Δ; −0.0181±0.29;and −0.0004±0.002, respectively. The 95% CI of the agreement between theHRs, kappa angles, and R² values estimated in two trials were ±1.57 Δ,±1.44Δ, and ±0.0098, respectively, as demonstrated in FIGS. 13A-13C.

Meanwhile, FIG. 14 shows a comparison of measured deviation (normalized)using the mobile device application and the ground truth for differentangles of fixation. Each point on the graph represents the meandeviation measured over all subjects, while the error bars represent thestandard deviation. The dashed line represents the regression line. Themobile device application measurements were normalized and scaled usingthe population average HR and HVID in order to scale the measurements toangular deviations. A regression analysis with the ground truth servingas an independent variable and the measured deviation as a dependentvariable led to the following results, as shown in FIG. 14: slope=1.0076(p<0.001); intercept=−0.1406 (p=0.48); and R²=0.998.

FIG. 15 shows the results of phoria measurements for 14 subjects withthe mobile device application and using a Modified Thorington (MT)method that is commonly used in clinics. As shown in FIG. 15, phoriameasurements of the mobile device application closely follow theclinical measurements (n=14; slope: 0.94; intercept: −1.12; R²=0.97;p<0.001). The overall RMSE between the application and MT method was1.7Δ. There was no correlation between the phoria and the difference inthe two measurement methods (r=−0.32; p=0.265). Within the studypopulation, more people had exophoria than esophoria. Within-subject MTmeasurements were within 1Δ for ten out of 14 subjects and within 2Δ for13 out of 14 subjects. The subject with largest difference between theapplication and MT (3.9Δ) also showed highest within-subject differencein MT measurement (5Δ). With the application, there was no significantdifference between trials within subjects (repeated measures ANOVA:F(2,41)=0.198; p=0.98). Overall with the application, the mean and 95%CI of the within-subject phoria measurement difference was −0.3Δ and±2.6Δ, respectively. In other words, the correlation between thedifference and the deviation measured with MT was not significant,indicating that there was no systematic error.

F. DISCUSSION

The results described above show that the measurements obtained by themobile device application described herein are accurate, repeatable, andcomparable to clinical measurements in normally sighted non-strabismicadults. Within-subject (or subject-specific) parameters such as the HRand kappa angle estimated by the two independent applicationmeasurements are consistent (the 95% CI of difference 1.54). A closelinear fit to the underlying eye deviation data to obtainsubject-specific parameters, such as the HR and kappa angle, isindicated by high R² values (≈0.99). Thus, the mobile device applicationdescribed herein can provide meaningful clinical information, as theresults indicate a strong correlation between such measurements andtraditional clinical methods, (e.g., MT).

With phoria measurements, the results show that the mobile deviceapplication provides meaningful clinical information. The resultsindicate a strong correlation between the application measurements andthe MT method. Furthermore, measuring phoria requires dissociationbetween the two eyes, and the video-based cover test mode of theapplication was ideal for conducting such measurements. The results alsosuggest that the same procedure can be employed for measuringintermittent strabismus.

The use of HVID allows for use of the iris center instead of pupilcenter as the reference for computing the corneal reflectiondecentration. The iris center can be a more robust reference formeasuring ocular alignment, especially when computing binoculardifferences. Another advantage of using iris center is that the limbusboundary is better delineated than the pupil boundary in visualspectrum, especially in case of people with dark irises. Notably, solong as the patient's eyes are symmetric, an off-center pupil will notaffect the strabismus results computed with the application.

Accordingly, techniques are described herein that allow for complexocular misalignment assessments using widely accessible mobile devices,such as a smart phone, tablet, and the like. The mobile device canleverage the native hardware already installed in many modem mobiledevices, such as a camera and a flash. Once a photograph of eyes of apatient is captured, the processor analyzes the photograph to detectvarious features in the patient's eyes and can then measure a magnitudeof ocular misalignment. The entire processing is performed on the mobiledevice itself, rather than sending the captured photographs to a remoteserver for processing.

Advantageously, the mobile application facilitates a quick, convenient,and inexpensive way to measure ocular misalignment, particularly in thecase of uncooperative patients such as non-verbal children or adultswith severe brain injuries. The ability to provide objectivemeasurements quickly and conveniently can be highly beneficial toclinicians who see large volumes of patients on a regular basis.Alternatively, the application can be used at home, for example, byparents who suspect their kids might have strabismus. Greateraccessibility to ocular screenings can help alleviate missed diagnosesin a case where a condition manifests intermittently. The application isalso well suited for telemedicine, which can be useful in remote,underserved areas, or for remote follow-up of treatment without thepatient having to visit the doctor. Further, due to the high-resolutioncapabilities of modern mobile device cameras, the application canmeasure small angle strabismus (e.g., less than 5 prism diopters (PD))that is otherwise difficult to detect using conventional clinicalmethods. Also, the application is robust in that it handles a variety ofimaging conditions and variability of eye appearance between subjects.

While there have been shown and described illustrative embodiments thatprovide for a mobile device application for ocular misalignmentmeasurement, it is to be understood that various other adaptations andmodifications may be made within the spirit and scope of the embodimentsherein. For instance, while a mobile device is frequently mentionedthroughout the present disclosure, the techniques described herein mayalso be implemented on desktop computers or similar machines. Thus, theembodiments of the present disclosure may be modified in any suitablemanner in accordance with the scope of the present claims.

The foregoing description has been directed to embodiments of thepresent disclosure. It will be apparent, however, that other variationsand modifications may be made to the described embodiments, with theattainment of some or all of their advantages. Accordingly, thisdescription is to be taken only by way of example and not to otherwiselimit the scope of the embodiments herein. Therefore, it is the objectof the appended claims to cover all such variations and modifications ascome within the true spirit and scope of the embodiments herein.

1-50. (canceled)
 51. A mobile device comprising: an image acquisitionunit configured to acquire an image of first and second eyes of apatient; a light source configured to provide light to reflect from anoptical surface of the first and second eyes of the patient; a processorconfigured to: obtain ocular misalignment measurements including atleast one of a magnitude and a direction of ocular misalignment in thefirst and second eyes of the patient; detect at least one of an iris anda pupil in the first and second eyes of the patient; calculate at leastone of an inter-pupillary distance and an iris diameter of the patient;and calculate a Hirschberg ratio of the patient based on the ocularmisalignment measurements.
 52. The device of claim 51, wherein theHirschberg ratio is further calculated based on at least one of theinter-pupillary distance and the iris diameter of the patient.
 53. Thedevice of claim 51, wherein the processor is further configured todetect a reflection on a corneal surface of the first eye of the patientand a reflection on a corneal surface of the second eye of the patientusing the acquired image, and configured to compare a position of thereflection in the first eye to a position of the reflection in thesecond eye.
 54. The device of claim 53, wherein the processor is furtherconfigured to: detect a reference point in the first eye and a referencepoint in the second eye using the acquired image; calculate a firstdistance between a position of the reference point in the first eye andthe position of the reflection in the first eye; calculate a seconddistance between a position of the reference point in the second eye andthe position of the reflection in the second eye; calculate a differencebetween the first distance and the second distance; and diagnose acondition of the patient based on the calculated difference between thefirst distance and the second distance, wherein the reference points inthe first and second eyes are a center of an iris or pupil of thepatient.
 55. The device of claim 54, wherein the processor is furtherconfigured to measure the positions of the reflections in the first andsecond eyes and the positions of the reference points in the first andsecond eyes in image space using the acquired image.
 56. The device ofclaim 55, wherein the processor is further configured to convert thefirst distance and the second distance into degrees or prism dioptersusing the Hirschberg ratio and an internal calibration factor based oniris diameter.
 57. The device of claim 54, wherein the processor isfurther configured to: compare the calculated difference between thefirst distance and the second distance to a predetermined misalignmentthreshold; diagnose the condition of the patient based on the comparisonof the calculated difference to the predetermined misalignmentthreshold; and determine that an ocular misalignment condition ispresent in the patient when the calculated difference is greater than orequal to the predetermined misalignment threshold.
 58. The device ofclaim 51, wherein the processor is further configured to detect areflection on a corneal surface of the first eye of the patient and areflection on a corneal surface of the second eye of the patient usingthe acquired image, and configured to compare a position of thereflection in the first eye to a position of the reflection in thesecond eye.
 59. The device of claim 51, wherein the processor is furtherconfigured to diagnose a condition of the patient or a risk factor ofthe patient developing a condition associated with ocular misalignmentbased on the ocular misalignment measurements.
 60. The device of claim51, wherein the processor is further configured to detect aniridoscleral border and a pupillary margin of the eyes of the patientusing the acquired image using an adaptive threshold algorithm.
 61. Ahand held device comprising: an image acquisition unit configured toacquire an image of first and second eyes of a patient; a light sourceconfigured to provide light to reflect from an optical surface of thefirst and second eyes of the patient; a processor configured to:determine ocular misalignment measurements of the first and second eyes,the ocular misalignment measurements including at least a magnitude anda direction of ocular misalignment in the first and second eyes of thepatient; measure a Kappa angle between an optical axis and a visual axisof the first and second eyes of the patient based on the acquired image;diagnose a condition of the patient based on the measured Kappa angle;and determining a deviation of the first and second eyes from the Kappaangle.
 62. The device of claim 61, wherein the processor is furtherconfigured to diagnose which eye of the first and second eyes of thepatient is misaligned and a misalignment direction of the misaligned eyebased on the Kappa angle, and to measure a fixation reliability index ofthe non-misaligned eye based on the deviation of the non-misaligned eyefrom the Kappa angle.
 63. The device of claim 61, wherein the processoris further configured to calculate at least one of an inter-pupillarydistance and an iris diameter of the patient; and perform at least onecalculation based on at least one of the ocular misalignmentmeasurements, the inter-pupillary distance, and the iris diameter of thepatient.
 64. The device of claim 61, wherein the processor is furtherconfigured to detect at least one of an iris and a pupil in the firstand second eyes of the patient.
 65. The device of claim 61, wherein theprocessor is further configured to calculate a Hirschberg ratio of thepatient.
 66. The device of claim 61, wherein the image acquisition unitis configured to acquire a plurality of images of the first and secondeyes of the patient; and wherein the processor is configured todetermine an image quality of the acquired plurality of images, andconfigured to use an image having a highest image quality among theacquired plurality of images when determine the ocular misalignmentmeasurements.
 67. The device of claim 61, wherein the image acquisitionunit is configured to acquire a plurality of images of the first andsecond eyes of the patient; and wherein the processor is configured toobtain the ocular misalignment measurements in each of the acquiredimages of the plurality of images, generate a composite of the obtainedocular misalignment measurements, and determine an overall ocularmeasurement based on the generated composite.
 68. The device of claim61, further comprising one or more motion sensors, wherein the processoris further configured to compensate for rotation of the device duringimage acquisition using measurements obtained by the motion sensors.